A neural network is a function built by composing one simple operation with itself: an affine map followed by a fixed non-linearity. Everything in this page — the representation theorems, the training algorithm, the architectures that read images and generate text — is a variation on that single composition. We build it from the linear models of the earlier pages, derive the backpropagation algorithm that trains it, and watch a small network learn a decision boundary that no linear model can draw.
What a network is, why depth is needed, and the non-linearity that makes depth worth anything.
For linear models, the target $Y$ (a number or a vector) is approximated by $$f(X) = W \cdot \hat{X}, \qquad \hat{X} = \begin{pmatrix} 1 \\ X \end{pmatrix},$$ where the leading $1$ absorbs the intercept into the matrix. A natural way to extend the linear model is to compose it with a non-linear function $\sigma$, applied entry-wise: $$g(X) = \sigma\big(W \cdot \hat{X}\big).$$ This is a single-layer neural network — the perceptron of the Classification page is its thresholded special case. Composing the construction with itself gives a multilayer network with $k$ hidden layers, $$X \mapsto \big(g_k \circ \cdots \circ g_2 \circ g_1 \circ g_0\big)(X), \qquad k \in \{0, 1, 2, \dots\},$$ where each $g_i(X) = \sigma(W_i \cdot \hat{X})$ has its own weight matrix. Training means choosing all the $W_i$ at once — by gradient descent on a loss, as derived below.
A single layer draws a single hyperplane. The classic witness is XOR: label the points $(0,0)$ and $(1,1)$ negative, $(1,0)$ and $(0,1)$ positive. No line separates the positive corners from the negative ones — the convex hulls of the two classes overlap — so no choice of $W$ makes a perceptron compute XOR. Minsky and Papert made this precise in 1969, and the field cooled for a decade. The fix is the hidden layer: with one hidden layer of two units, XOR is a four-line construction — each hidden unit draws one line, and the output unit combines the two half-planes into a stripe.
Depth is not merely sufficient; it is efficient. One hidden layer already gives universality:
The theorem says width can buy anything — but possibly at exponential width. Depth buys the same functions cheaper: there are functions representable by a depth-$k$ network of polynomial size whose shallow approximations need exponentially many units (a counting argument on the number of linear regions of a ReLU network makes this concrete — depth multiplies regions, width only adds them). Universality is an existence statement, though: it promises a good network exists, not that gradient descent will find it. That gap between representation and optimization is where the rest of the subject lives.
The activation $\sigma$ is the only source of non-linearity: without it, a stack of affine maps collapses to a single affine map, and depth buys nothing. Three demands shape the choice — $\sigma$ must be non-linear, (almost everywhere) differentiable so gradients can flow, and cheap to evaluate billions of times.
The sigmoid was the default for decades: it squashes $\mathbb{R}$ to $(0,1)$, which reads naturally as a probability, and its derivative $\sigma(1-\sigma)$ is computable from its own value. Its weakness is the tails — where $|\sigma'| \le \frac14$ and decays exponentially, gradients shrink through every saturated layer, the vanishing gradient problem. tanh is the zero-centred repair (range $(-1,1)$, derivative up to $1$) but saturates the same way. The modern default is the ReLU: the identity on the positive half-line, so gradients pass through undiminished, and a max on the negative half — at the price of units that can switch off permanently ("dying ReLUs"), patched by the leaky variant $\max(\alpha x, x)$, $\alpha \approx 0.01$. At the output layer the activation follows the task: sigmoid for a probability, softmax for a distribution over classes, the identity for regression.
How a network is trained: the loss it minimizes, the chain-rule computation that makes minimizing it cheap, the algorithm at work on a live decision boundary, and what it takes to train well in practice.
Training is empirical risk minimization. Given data $(x_i, y_i)$, $i = 1, \dots, m$, and parameters $\theta$ collecting every $W^{(l)}$ and $b^{(l)}$, minimize $$L(\theta) = \frac{1}{m} \sum_{i=1}^{m} \ell\big(f_\theta(x_i), y_i\big),$$ with $\ell$ matched to the task — squared error for regression, and for classification the cross-entropy $\ell(p, y) = -\big[y \log p + (1-y) \log(1-p)\big]$, whose gradient with a sigmoid output collapses beautifully to $p - y$. The workhorse is gradient descent, $\theta \leftarrow \theta - \eta \nabla_\theta L$, usually on minibatches (stochastic gradient descent) — the Optimization page studies the step itself. What a neural network adds is a way to compute $\nabla_\theta L$ for $10^9$ parameters at the cost of roughly two function evaluations. That computation is backpropagation.
Write the forward pass with pre-activations $z^{(l)}$: $$z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}, \qquad a^{(l)} = \sigma\big(z^{(l)}\big), \qquad a^{(0)} = x.$$ Define the error at layer $l$ as the gradient of the loss with respect to that layer's pre-activations, $\delta^{(l)} = \partial L / \partial z^{(l)}$. Two applications of the chain rule give everything. At the output layer the error is direct: $$\delta^{(L)} = \nabla_{a^{(L)}} L \odot \sigma'\big(z^{(L)}\big),$$ and each earlier layer inherits its error from the next one, transported backwards through the weight matrix and re-weighted by the local derivative: $$\delta^{(l)} = \Big(\big(W^{(l+1)}\big)^{\!\top} \delta^{(l+1)}\Big) \odot \sigma'\big(z^{(l)}\big).$$ The parameter gradients are then outer products of errors with the activations that caused them: $$\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} \big(a^{(l-1)}\big)^{\!\top}, \qquad \frac{\partial L}{\partial b^{(l)}} = \delta^{(l)}.$$
Three observations. First, the algorithm is just the chain rule with bookkeeping — its content is the reuse: every $\delta^{(l)}$ is computed once and used twice, so the backward pass costs about as much as the forward pass regardless of the number of parameters; finite differences would cost one forward pass per parameter. Second, the recursion explains the failure modes described with the activations: $\sigma'$ appears as a factor at every layer, so saturating activations multiply the gradient by small numbers $L$ times. Third, nothing in the derivation is specific to dense layers — any computation graph can be differentiated this way, which is why modern frameworks (PyTorch, JAX) implement reverse-mode automatic differentiation over arbitrary programs, and backpropagation is its feedforward special case. The derivation was popularized by Rumelhart, Hinton and Williams (1986); the idea itself is older, appearing in control theory and in Werbos's 1974 thesis.
Theory into practice: a $2 \to H \to 1$ network — tanh hidden units, sigmoid output, cross-entropy loss, plain gradient descent — trained by exactly the recursion above, implemented in a few dozen lines of JavaScript with no library. XOR is the dataset a perceptron cannot fit; the two moons and the ring are its curved cousins. Watch the boundary bend: the hidden units are drawing lines, and the output unit is combining them into regions no single line could make.
Initialization. The backward recursion multiplies by $\sigma'$ and by the weights layer after layer, so the scale of $W$ decides whether signals explode or vanish exponentially in depth. The repair is to fix the variance at initialization: draw weights with $\mathrm{Var}(W) = \frac{2}{n_{in} + n_{out}}$ (Xavier/Glorot, right for tanh and sigmoid) or $\frac{2}{n_{in}}$ (He, right for ReLU, whose positive half discards half the variance). Symmetry is broken by randomness itself — identical initial weights would keep hidden units identical forever.
Learning rate. The step size $\eta$ is the hyperparameter that matters most: too small and training crawls; too large and the loss oscillates or diverges, since gradient descent is only guaranteed to descend for $\eta$ below the inverse curvature of the loss surface. Schedules (decay, warmup) and adaptive methods (Adam and its relatives, which rescale each coordinate by a running average of its gradient magnitudes) are the standard mitigations.
Overfitting. A network with more parameters than data can interpolate the training set, noise included. The defences: more data; weight decay, which adds $\frac{\lambda}{2}\|\theta\|^2$ to the loss and pulls unused weights to zero; dropout, which trains a random sub-network at each step and averages exponentially many of them at test time; and early stopping, which treats the epoch count itself as a regularizer. The modern surprise — very large networks that interpolate their training data yet still generalize, the double-descent phenomenon — is real, but not a licence to skip the defences.
Images break the dense-layer template: a $256 \times 256$ photograph has $65{,}536$ inputs, and a dense first layer with $1{,}000$ units spends $65$ million parameters learning that a cat's ear is the same pattern wherever it appears. A convolutional layer builds the prior in: each unit looks at a small patch of the previous layer (locality), every unit at a position uses the same weights (weight sharing), so the layer computes a bank of learned pattern detectors slid across the image — a feature that appears can be detected anywhere (translation equivariance). Pooling then downsamples, trading position for invariance. The detectors themselves are small matrices called kernels, and the classical image-processing kernels below are exactly the hand-designed ancestors of what a CNN learns from data.
Draw a digit on the pad. A small convolutional network, trained on MNIST and running entirely in this page, reads it and reports its ten class probabilities.
Three architecture families that reuse the same feedforward machinery for other jobs: compression, sequences, and generation.
An autoencoder trains a network to copy its input through a narrow bottleneck: an encoder $E$ compresses $x$ to a code $z = E(x)$ of much lower dimension, a decoder $D$ reconstructs $\hat{x} = D(z)$, and the loss $\|x - \hat{x}\|^2$ forces the code to keep what matters. With linear activations the optimal code spans the top principal components — an autoencoder is non-linear PCA — and the code layer, not the reconstruction, is usually the point: a learned coordinate system for the data.
Sequence models before 2017 read one token at a time; the transformer reads them all at once and lets each position look at every other through attention: with queries $Q$, keys $K$, values $V$ computed from the inputs, $$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{Q K^{\top}}{\sqrt{d_k}}\right) V,$$ a differentiable dictionary lookup — each query retrieves a weighted average of the values, weighted by query–key agreement, with the $1/\sqrt{d_k}$ keeping the logits at unit scale. Stacked in blocks with feedforward layers and residual connections, attention turns out to be sufficient ("Attention is all you need", Vaswani et al., 2017): transformers now underlie essentially all large language models and most of vision as well. The cost is quadratic in sequence length; much current research is spent buying that factor down.
A diffusion model learns to generate by learning to undo noise. The forward process is fixed: gradually corrupt data $x_0$ with Gaussian noise, $q(x_t \mid x_{t-1}) = \mathcal{N}\big(\sqrt{1-\beta_t}\, x_{t-1},\, \beta_t I\big)$, until $x_T$ is pure static. The model learns the reverse steps $p_\theta(x_{t-1} \mid x_t)$ — in practice it learns to predict the noise that was added, equivalently the score $\nabla \log p_t(x)$ — and generation is the reverse walk: sample static, then denoise it, step by step, into a photograph, a sentence's embedding, a protein. The same score function connects diffusion to Langevin dynamics and to the energy-based models of statistical physics (Ho, Jain and Abbeel, 2020).